Web Scraping Whoopsies

Manu Alcalá + Jameson Carter

2023-10-18

What is web scraping?

Broadly, it is the process of automatically pulling data from websites by reading their underlying code. Doing this gets complicated fast:

Static websites contain information in HTML code which does not change when you, the user, interact with it.
Dynamic websites have information in HTML code which does change as you interact with it.
Static vs. Dynamic websites require different code packages to scrape.
- For example, selenium is a common dynamic web scraping library and scrapy is a common static web scraping library.
- This is because a static scraper simply has to understand what it is looking for in the HTML. A dynamic scraper needs to simulate a human interacting with the site.

Generally, web scraping is the process of automatically reading the content of websites, in our case to get data. Within a website’s underlying code, there might be something that you want to access. For example, a large table published on a website with no download button. But getting that hypothetical table depends on how the website is set up. Static sites are relatively easy to scrape because all of the information is available to you through the code which built the site, at all times. Dynamic sites work a little differently: they require you, the user, to click on buttons and interact with the site.

Because of this, the tools we use differ. Static scrapers usually pull the sites’ code directly and then parse that code to extract relevant information. Dynamic scrapers simulate a human being by clicking on the right series of buttons, to trick the site into providing its data, which can then be parsed. You will hear us mention both scrapy and selenium today. Selenium is used for dynamic sites and scrapy is used for static sites .

When is it worth it?

Other data collection efforts are impossible or prone to human error
- Creating reproducible and reliable processes eases collaboration / quality assurance
Getting new data releases quickly is valuable
Time spent coding < time spent obtaining data in other ways + time spent on quality assurance

As you might have guessed, this takes some time. Before getting into it, you should consider whether other methods of data collection are error-prone or so daunting that you could never justify doing it by hand. You have emailed the owner of the data to no avail and you fear that manually copying 1000 entries could be subject to error (or a waste of time). Also, it is easier to collaborate with code-based processes, explaining how you hand-generated tables can sometimes feel like more trouble than its worth!

Additionally, consider whether getting new data quickly is valuable. For example, if a government site releases data regularly you may want to have a process ready to extract those data. Web scraping is great for this.

Finally, note that when planning these things it might seem like a bunch of code on the front-end. But if the time spent coding is less than time spent obtaining data in other ways PLUS time spent on quality assurance of those methods, it could be well worth it! Validating processes is easier with code.

What makes it hard?

It is difficult to predict how much time a web scraping task will take.
Sites might change, introducing need to update.
- Sites might be removed, stop being maintained, or completely overhauled.
Site maintainers may not be okay with data being scraped. Quick plug for the TECH team’s Automated Data Guidelines.

Example 1. Scraping Medicaid enrollment data

Why Medicaid enrollment data?

Since Spring 2023, states have been dis-enrolling Medicaid beneficiaries who no longer qualify since the Public Health Emergency was ended.

Why are the data interesting?

In anticipation of “the great unwinding,” many states implemented policy changes to smooth the transition.

To understand the success of these policies, we wanted time-series enrollment data for all 50 states… from a Medicaid data system that is largely decentralized.

Unreadable PDFs abound!

A sigh of relief…

Why page through PDFs when another organization’s RAs can do it for you?

1. Identify this is a scrape-able dynamic page

One URL with data you can only get by clicking each option!

2. Confirm HTML actually contains the data

3. Code for 30 hours!

4. Bask in the glow of automated scraping

Whenever new data were released in the following 2 months, I re-ran the code and got a well-formatted excel file as output.

Little did I know, trouble was coming

What happened?

2 months later, KFF stopped updating the dashboard and changed how existing data was reported on graphs.

Example 2. Scraping Course Descriptions

Work Based Learning (WBL)

Work-based learning can include internships, clinicals, co-ops and other opportunities to gain experience in a work setting.

They are especially helpful to community college students

We sought to create a national dataset of WBL prevalance by scraping course descriptions

Course description data

No centralized source, need to scrape each school individually
No general pattern to where course descriptions can be found, how they are arranged, or what format they’re in

An example of course descriptions listed under department pages

And an example containing links to course catalogs in .pdf format

Are we doomed?

How do we find all the relevant links?
How can we target HTML code when each website has an entirely different one?

Web Crawling with Scrapy

Scrapy is a web scraping framework designed for large-scale web scraping projects
It excels particularly in web crawling, i.e. traversing links and navigating websites in accordance with specified rules.
It has several helpful features to respect scraping restrictions, avoid overwhelming servers, and pre-process scraped data

Initial Web Scraping Attempt

Starting from a homepage, we used Scrapy to follow links containing keywords like “catalog” or “course descriptions”

For each link, we scraped basic metadata and all the text present

Whoopsie…

After a lot of hard work refining our approach, our data was a hot mess:

We did not get any data from about 1/3 of the schools
We couldn’t uniquely identify courses which made it impossible to filter out duplicates
We couldn’t verify whether we had found all course descriptions

A different approach

We decided to lower our ambition and hone-in on a single state instead

Disaster Averted

We used Selenium to navigate to each course until finding a static page containing course descriptions.
We used BeautifulSoup next to parse specific HTML tags (which were standard across sites)

Concluding remarks

Core questions to explore before you start

Availability of data

Are the data available through other routes?
Are the data produced by an organization that is invested in the problem long-term?

Frequency of scraping

Will I need to scrape the data multiple times?
What is the risk that the item scraped from the site will be changed?

Time-value tradeoffs

Is the time spent coding worth the payoff?
Will collecting data automatically save time on quality assurance?

Questions?

The remainder of the time is reserved for group discussion!

Thank you!

Please contact Manu Alcalá or Jameson Carter if you would like to discuss either of these projects or scope whether a use-case is reasonable.